class: center, middle, inverse, title-slide .title[ # <span style="font-size:48pt;">Cross-Validation: Advanced</span> ] .subtitle[ ## 📚 ⚒️ 🤓️ ] .author[ ### Pittsburgh Summer Methodology Series ] .date[ ### Day 2C     August 9, 2022 ] --- class: inverse, center, middle # Overview --- class: onecol ## Motivation Yesterday, we introduced the rationale for **holdout cross-validation**. -- <img src="data:image/png;base64,#../figs/holdout.png" width="70%" /> -- This is a good first step! However, we often want model performance *before getting to the final test set*. --- class: onecol ## Plan for Today Today's focus is on advanced cross-validation methods, including: - *k*-fold cross-validation - repeated *k*-fold cross-validation - leave-one-out cross-validation (LOOCV) - nested cross-validation -- All CV methods will be performed with {rsample} and {tune}. <img src="data:image/png;base64,#../figs/rsample_tune.png" width="30%" /> --- class: inverse, center, middle # k-fold Cross-Validation --- ## k-fold Cross-Validation <br> <img src="data:image/png;base64,#../figs/kfold1.png" width="95%" /> --- count: false ## k-fold Cross-Validation .center[ <img src="data:image/png;base64,#../figs/kfold2.png" width="70%" /> ] --- class: onecol ## k-fold Cross-Validation The final resampling performance estimate is averaged across each *k* fold. k-fold cross-validation can also be .imp[stratified] to keep the sets relatively similar. k-fold cross-validation can also be .imp[repeated] to avoid problems with any single split. -- <p style="padding-top:30px;">How many folds should be used in cross-validation? Larger values of *k* result in resampling estimates with **lower bias** but **higher variance**. Smaller values of *k* result in estimates with **higher bias** but **lower variance**. -- .bg-light-green.b--dark-green.ba.bw1.br3.pl4[ **Advice**: In practice, values of *k* = 5 or *k* = 10 are most common. ] --- class: onecol ## k-fold Cross-Validation in R We will use the `vfold_cv()` function from {rsample}¹. Argument | Description :------- | :---------- data | The data frame v | The number of partitions to create (default = 10) repeats | The number of times to repeat the v-fold partitioning (default = 1) strata | Variable to conduct stratified sampling (default = NULL) .footnote[ [1] Note that {tidymodels} refers to k-fold as v-fold CV. These refer to the same resampling process! ] --- class: onecol ## k-fold Cross-Validation in R Let's perform 10-fold cross-validation repeated 3 times using the `titanic` dataset. Our goal will be to predict each passenger's fare (how much they paid). -- .scroll40[ ```r library(tidymodels) titanic <- read.csv("https://tinyurl.com/titanic-pm") set.seed(2022) fare_folds <- vfold_cv(titanic, v = 10, repeats = 3, strata = fare) fare_folds #> # 10-fold cross-validation repeated 3 times using stratification #> # A tibble: 30 × 3 #> splits id id2 #> <list> <chr> <chr> #> 1 <split [1177/132]> Repeat1 Fold01 #> 2 <split [1177/132]> Repeat1 Fold02 #> 3 <split [1177/132]> Repeat1 Fold03 #> 4 <split [1177/132]> Repeat1 Fold04 #> 5 <split [1178/131]> Repeat1 Fold05 #> 6 <split [1178/131]> Repeat1 Fold06 #> 7 <split [1178/131]> Repeat1 Fold07 #> 8 <split [1179/130]> Repeat1 Fold08 #> 9 <split [1180/129]> Repeat1 Fold09 #> 10 <split [1180/129]> Repeat1 Fold10 #> # … with 20 more rows ``` ] -- The `splits` column contains information on how to split the data. `[297/33]` indicates N = 297 in the analysis set and N = 33 in that particular k-fold. --- class: onecol ## k-fold Cross-Validation in R We can extract individual resampled data folds with `analysis()` and `assessment()`. -- .scroll30[ ```r fare_folds$splits[[1]] %>% analysis() #> survived pclass sex age sibsp parch fare #> 1 1 1 female 29.0000 0 0 211.3375 #> 3 0 1 female 2.0000 1 2 151.5500 #> 4 0 1 male 30.0000 1 2 151.5500 #> 5 0 1 female 25.0000 1 2 151.5500 #> 6 1 1 male 48.0000 0 0 26.5500 #> 7 1 1 female 63.0000 1 0 77.9583 #> 8 0 1 male 39.0000 0 0 0.0000 #> 9 1 1 female 53.0000 2 0 51.4792 #> 10 0 1 male 71.0000 0 0 49.5042 #> 11 0 1 male 47.0000 1 0 227.5250 #> 12 1 1 female 18.0000 1 0 227.5250 #> 13 1 1 female 24.0000 0 0 69.3000 #> 15 1 1 male 80.0000 0 0 30.0000 #> 17 0 1 male 24.0000 0 1 247.5208 #> 18 1 1 female 50.0000 0 1 247.5208 #> 21 1 1 male 37.0000 1 1 52.5542 #> 22 1 1 female 47.0000 1 1 52.5542 #> 23 1 1 male 26.0000 0 0 30.0000 #> 24 1 1 female 42.0000 0 0 227.5250 #> 26 0 1 male 25.0000 0 0 26.0000 #> 27 1 1 male 25.0000 1 0 91.0792 #> 28 1 1 female 19.0000 1 0 91.0792 #> 29 1 1 female 35.0000 0 0 135.6333 #> 30 1 1 male 28.0000 0 0 26.5500 #> 31 0 1 male 45.0000 0 0 35.5000 #> 32 1 1 male 40.0000 0 0 31.0000 #> 33 1 1 female 30.0000 0 0 164.8667 #> 35 0 1 male 42.0000 0 0 26.5500 #> 37 1 1 female 22.0000 0 1 55.0000 #> 38 1 1 male NA 0 0 26.5500 #> 39 0 1 male 41.0000 0 0 30.5000 #> 40 0 1 male 48.0000 0 0 50.4958 #> 42 1 1 female 44.0000 0 0 27.7208 #> 43 1 1 female 59.0000 2 0 51.4792 #> 44 1 1 female 60.0000 0 0 76.2917 #> 45 1 1 female 41.0000 0 0 134.5000 #> 47 0 1 male NA 0 0 31.0000 #> 48 1 1 male 42.0000 0 0 26.2875 #> 49 1 1 female 53.0000 0 0 27.4458 #> 50 1 1 male 36.0000 0 1 512.3292 #> 51 1 1 female 58.0000 0 1 512.3292 #> 54 0 1 male 17.0000 0 0 47.1000 #> 55 1 1 male 11.0000 1 2 120.0000 #> 56 1 1 female 14.0000 1 2 120.0000 #> 57 1 1 male 36.0000 1 2 120.0000 #> 58 1 1 female 36.0000 1 2 120.0000 #> 59 0 1 male 49.0000 0 0 26.0000 #> 60 1 1 female NA 0 0 27.7208 #> 61 0 1 male 36.0000 1 0 78.8500 #> 62 1 1 female 76.0000 1 0 78.8500 #> 64 1 1 female 47.0000 1 0 61.1750 #> 65 1 1 male 27.0000 1 0 53.1000 #> 67 1 1 female 36.0000 0 0 262.3750 #> 68 1 1 female 30.0000 0 0 86.5000 #> 69 1 1 male 45.0000 0 0 29.7000 #> 72 0 1 male 27.0000 1 0 136.7792 #> 73 1 1 female 26.0000 1 0 136.7792 #> 75 0 1 male NA 0 0 52.0000 #> 76 0 1 male 47.0000 0 0 25.5875 #> 77 1 1 female 39.0000 1 1 83.1583 #> 78 0 1 male 37.0000 1 1 83.1583 #> 79 1 1 female 64.0000 0 2 83.1583 #> 81 0 1 male NA 0 0 26.5500 #> 82 0 1 male 70.0000 1 1 71.0000 #> 83 1 1 female 36.0000 0 2 71.0000 #> 84 1 1 female 64.0000 1 1 26.5500 #> 85 0 1 male 39.0000 1 0 71.2833 #> 86 1 1 female 38.0000 1 0 71.2833 #> 87 1 1 male 51.0000 0 0 26.5500 #> 88 1 1 male 27.0000 0 0 30.5000 #> 89 1 1 female 33.0000 0 0 151.5500 #> 90 0 1 male 31.0000 1 0 52.0000 #> 91 1 1 female 27.0000 1 2 52.0000 #> 92 1 1 male 31.0000 1 0 57.0000 #> 93 1 1 female 17.0000 1 0 57.0000 #> 94 1 1 male 53.0000 1 1 81.8583 #> 95 1 1 male 4.0000 0 2 81.8583 #> 96 1 1 female 54.0000 1 1 81.8583 #> 97 0 1 male 50.0000 1 0 106.4250 #> 98 1 1 female 27.0000 1 1 247.5208 #> 99 1 1 female 48.0000 1 0 106.4250 #> 100 1 1 female 48.0000 1 0 39.6000 #> 101 1 1 male 49.0000 1 0 56.9292 #> 102 0 1 male 39.0000 0 0 29.7000 #> 103 1 1 female 23.0000 0 1 83.1583 #> 104 1 1 female 38.0000 0 0 227.5250 #> 105 1 1 female 54.0000 1 0 78.2667 #> 106 0 1 female 36.0000 0 0 31.6792 #> 107 0 1 male NA 0 0 221.7792 #> 108 1 1 female NA 0 0 31.6833 #> 109 1 1 female NA 0 0 110.8833 #> 111 0 1 male 30.0000 0 0 27.7500 #> 112 1 1 female 24.0000 3 2 263.0000 #> 113 1 1 female 28.0000 3 2 263.0000 #> 114 1 1 female 23.0000 3 2 263.0000 #> 115 0 1 male 19.0000 3 2 263.0000 #> 116 0 1 male 64.0000 1 4 263.0000 #> 117 1 1 female 60.0000 1 4 263.0000 #> 119 0 1 male NA 0 0 26.5500 #> 120 1 1 male 50.0000 2 0 133.6500 #> 121 1 1 male 43.0000 1 0 27.7208 #> 122 1 1 female NA 1 0 133.6500 #> 123 1 1 female 22.0000 0 2 49.5000 #> 124 1 1 male 60.0000 1 1 79.2000 #> 125 1 1 female 48.0000 1 1 79.2000 #> 126 0 1 male NA 0 0 0.0000 #> 127 0 1 male 37.0000 1 0 53.1000 #> 128 1 1 female 35.0000 1 0 53.1000 #> 130 1 1 female 35.0000 0 0 211.5000 #> 131 1 1 female 22.0000 0 1 59.4000 #> 133 0 1 male 24.0000 0 0 79.2000 #> 134 1 1 male 49.0000 1 0 89.1042 #> 135 1 1 female NA 1 0 89.1042 #> 136 0 1 male 71.0000 0 0 34.6542 #> 138 1 1 female 19.0000 0 0 30.0000 #> 139 0 1 male 38.0000 0 1 153.4625 #> 140 1 1 female 58.0000 0 1 153.4625 #> 141 1 1 male 23.0000 0 1 63.3583 #> 143 0 1 male 46.0000 0 0 79.2000 #> 144 1 1 male 25.0000 1 0 55.4417 #> 145 1 1 female 25.0000 1 0 55.4417 #> 147 1 1 female 49.0000 1 0 76.7292 #> 149 0 1 male 45.0000 1 0 83.4750 #> 150 1 1 female 35.0000 1 0 83.4750 #> 151 0 1 male 40.0000 0 0 0.0000 #> 153 1 1 male NA 0 0 30.0000 #> 154 1 1 female 24.0000 0 0 83.1583 #> 155 0 1 male 55.0000 1 1 93.5000 #> 156 1 1 female 52.0000 1 1 93.5000 #> 157 0 1 male 42.0000 0 0 42.5000 #> 158 0 1 male NA 0 0 51.8625 #> 159 0 1 male 55.0000 0 0 50.0000 #> 160 1 1 female 16.0000 0 1 57.9792 #> 161 1 1 female 44.0000 0 1 57.9792 #> 162 1 1 female 51.0000 1 0 77.9583 #> 163 0 1 male 42.0000 1 0 52.0000 #> 164 1 1 female 35.0000 1 0 52.0000 #> 165 1 1 male 35.0000 0 0 26.5500 #> 166 1 1 male 38.0000 1 0 90.0000 #> 167 0 1 male NA 0 0 30.6958 #> 168 1 1 female 35.0000 1 0 90.0000 #> 169 1 1 female 38.0000 0 0 80.0000 #> 170 0 1 female 50.0000 0 0 28.7125 #> 172 0 1 male 46.0000 0 0 26.0000 #> 173 0 1 male 50.0000 0 0 26.0000 #> 174 0 1 male 32.5000 0 0 211.5000 #> 175 0 1 male 58.0000 0 0 29.7000 #> 176 0 1 male 41.0000 1 0 51.8625 #> 177 1 1 female NA 1 0 51.8625 #> 178 1 1 male 42.0000 1 0 52.5542 #> 179 1 1 female 45.0000 1 0 52.5542 #> 181 1 1 female 39.0000 0 0 211.3375 #> 182 1 1 female 49.0000 0 0 25.9292 #> 183 1 1 female 30.0000 0 0 106.4250 #> 184 1 1 male 35.0000 0 0 512.3292 #> 185 0 1 male NA 0 0 27.7208 #> 186 0 1 male 42.0000 0 0 26.5500 #> 187 1 1 female 55.0000 0 0 27.7208 #> 188 1 1 female 16.0000 0 1 39.4000 #> 189 1 1 female 51.0000 0 1 39.4000 #> 190 0 1 male 29.0000 0 0 30.0000 #> 191 1 1 female 21.0000 0 0 77.9583 #> 192 0 1 male 30.0000 0 0 45.5000 #> 193 1 1 female 58.0000 0 0 146.5208 #> 194 1 1 female 15.0000 0 1 211.3375 #> 195 0 1 male 30.0000 0 0 26.0000 #> 196 1 1 female 16.0000 0 0 86.5000 #> 197 1 1 male NA 0 0 29.7000 #> 199 1 1 female 18.0000 1 0 53.1000 #> 200 1 1 female 24.0000 0 0 49.5042 #> 201 0 1 male 46.0000 0 0 75.2417 #> 202 0 1 male 54.0000 0 0 51.8625 #> 203 1 1 male 36.0000 0 0 26.2875 #> 204 0 1 male 28.0000 1 0 82.1708 #> 205 1 1 female NA 1 0 82.1708 #> 207 0 1 male 44.0000 2 0 90.0000 #> 208 1 1 female 33.0000 1 0 90.0000 #> 209 1 1 female 37.0000 1 0 90.0000 #> 210 1 1 male 30.0000 1 0 57.7500 #> 211 0 1 male 55.0000 0 0 30.5000 #> 213 0 1 male 37.0000 0 1 29.7000 #> 214 1 1 female 31.0000 1 0 113.2750 #> 215 1 1 female 23.0000 1 0 113.2750 #> 216 0 1 male 58.0000 0 2 113.2750 #> 217 1 1 female 19.0000 0 2 26.2833 #> 218 0 1 male 64.0000 0 0 26.0000 #> 219 1 1 female 39.0000 0 0 108.9000 #> 220 1 1 male NA 0 0 25.7417 #> 221 1 1 female 22.0000 0 1 61.9792 #> 222 0 1 male 65.0000 0 1 61.9792 #> 223 0 1 male 28.5000 0 0 27.7208 #> 224 0 1 male NA 0 0 0.0000 #> 225 0 1 male 45.5000 0 0 28.5000 #> 226 0 1 male 23.0000 0 0 93.5000 #> 227 0 1 male 29.0000 1 0 66.6000 #> 228 1 1 female 22.0000 1 0 66.6000 #> 229 0 1 male 18.0000 1 0 108.9000 #> 230 1 1 female 17.0000 1 0 108.9000 #> 233 0 1 male 47.0000 0 0 52.0000 #> 234 1 1 female 56.0000 0 1 83.1583 #> 235 0 1 male 38.0000 0 0 0.0000 #> 236 1 1 male NA 0 0 39.6000 #> 237 0 1 male 22.0000 0 0 135.6333 #> 238 0 1 male NA 0 0 227.5250 #> 239 1 1 female 43.0000 0 1 211.3375 #> 240 0 1 male 31.0000 0 0 50.4958 #> 241 1 1 male 45.0000 0 0 26.5500 #> 242 0 1 male NA 0 0 50.0000 #> 243 1 1 female 33.0000 0 0 27.7208 #> 244 0 1 male 46.0000 0 0 79.2000 #> 245 0 1 male 36.0000 0 0 40.1250 #> 247 0 1 male 55.0000 1 0 59.4000 #> 248 1 1 female 54.0000 1 0 59.4000 #> 249 0 1 male 33.0000 0 0 26.5500 #> 250 1 1 male 13.0000 2 2 262.3750 #> 251 1 1 female 18.0000 2 2 262.3750 #> 252 1 1 female 21.0000 2 2 262.3750 #> 253 0 1 male 61.0000 1 3 262.3750 #> 254 1 1 female 48.0000 1 3 262.3750 #> 255 1 1 male NA 0 0 30.5000 #> 256 1 1 female 24.0000 0 0 69.3000 #> 257 1 1 male NA 0 0 26.0000 #> 258 1 1 female 35.0000 1 0 57.7500 #> 259 1 1 female 30.0000 0 0 31.0000 #> 260 1 1 male 34.0000 0 0 26.5500 #> 261 1 1 female 40.0000 0 0 153.4625 #> 263 0 1 male 50.0000 1 0 55.9000 #> 264 1 1 female 39.0000 1 0 55.9000 #> 265 1 1 male 56.0000 0 0 35.5000 #> 266 1 1 male 28.0000 0 0 35.5000 #> 267 0 1 male 56.0000 0 0 26.5500 #> 268 0 1 male 56.0000 0 0 30.6958 #> 269 0 1 male 24.0000 1 0 60.0000 #> 271 1 1 female 18.0000 1 0 60.0000 #> 272 1 1 male 24.0000 1 0 82.2667 #> 273 1 1 female 23.0000 1 0 82.2667 #> 274 1 1 male 6.0000 0 2 134.5000 #> 275 1 1 male 45.0000 1 1 134.5000 #> 276 1 1 female 40.0000 1 1 134.5000 #> 277 0 1 male 57.0000 1 0 146.5208 #> 278 1 1 female NA 1 0 146.5208 #> 279 1 1 male 32.0000 0 0 30.5000 #> 280 0 1 male 62.0000 0 0 26.5500 #> 281 1 1 male 54.0000 1 0 55.4417 #> 282 1 1 female 43.0000 1 0 55.4417 #> 283 1 1 female 52.0000 1 0 78.2667 #> 284 0 1 male NA 0 0 27.7208 #> 285 1 1 female 62.0000 0 0 80.0000 #> 287 0 1 female 63.0000 1 0 221.7792 #> 288 0 1 male 61.0000 0 0 32.3208 #> 289 1 1 female 48.0000 0 0 25.9292 #> 290 1 1 female 18.0000 0 2 79.6500 #> 291 0 1 male 52.0000 1 1 79.6500 #> 293 1 1 male 48.0000 1 0 52.0000 #> 294 1 1 female NA 1 0 52.0000 #> 295 0 1 male 49.0000 1 1 110.8833 #> 297 1 1 female 39.0000 1 1 110.8833 #> 298 1 1 female NA 0 0 79.2000 #> 299 1 1 male 31.0000 0 0 28.5375 #> 300 0 1 male 40.0000 0 0 27.7208 #> 301 0 1 male 61.0000 0 0 33.5000 #> 302 0 1 male 47.0000 0 0 34.0208 #> 303 1 1 female 35.0000 0 0 512.3292 #> 304 0 1 male 64.0000 1 0 75.2500 #> 305 1 1 female 60.0000 1 0 75.2500 #> 306 0 1 male 60.0000 0 0 26.5500 #> 307 0 1 male 54.0000 0 1 77.2875 #> 308 0 1 male 21.0000 0 1 77.2875 #> 309 1 1 female 55.0000 0 0 135.6333 #> 310 1 1 female 31.0000 0 2 164.8667 #> 311 0 1 male 57.0000 1 1 164.8667 #> 312 1 1 female 45.0000 1 1 164.8667 #> 313 0 1 male 50.0000 1 1 211.5000 #> 314 0 1 male 27.0000 0 2 211.5000 #> 315 1 1 female 50.0000 1 1 211.5000 #> 316 1 1 female 21.0000 0 0 26.5500 #> 318 1 1 male 21.0000 0 1 61.3792 #> 319 0 1 male NA 0 0 35.0000 #> 320 1 1 female 31.0000 0 0 134.5000 #> 322 0 1 male 62.0000 0 0 26.5500 #> 323 1 1 female 36.0000 0 0 135.6333 #> 324 0 2 male 30.0000 1 0 24.0000 #> 326 0 2 male 30.0000 0 0 13.0000 #> 327 0 2 male 18.0000 0 0 11.5000 #> 328 0 2 male 25.0000 0 0 10.5000 #> 329 0 2 male 34.0000 1 0 26.0000 #> 330 1 2 female 36.0000 1 0 26.0000 #> 331 0 2 male 57.0000 0 0 13.0000 #> 332 0 2 male 18.0000 0 0 11.5000 #> 333 0 2 male 23.0000 0 0 10.5000 #> 334 1 2 female 36.0000 0 0 13.0000 #> 335 0 2 male 28.0000 0 0 10.5000 #> 337 1 2 male 32.0000 1 0 26.0000 #> 338 1 2 female 19.0000 1 0 26.0000 #> 339 0 2 male 28.0000 0 0 26.0000 #> 340 1 2 male 1.0000 2 1 39.0000 #> 341 1 2 female 4.0000 2 1 39.0000 #> 342 1 2 female 12.0000 2 1 39.0000 #> 343 1 2 female 36.0000 0 3 39.0000 #> 344 1 2 male 34.0000 0 0 13.0000 #> 345 1 2 female 19.0000 0 0 13.0000 #> 346 0 2 male 23.0000 0 0 13.0000 #> 347 0 2 male 26.0000 0 0 13.0000 #> 348 0 2 male 42.0000 0 0 13.0000 #> 349 0 2 male 27.0000 0 0 13.0000 #> 350 1 2 female 24.0000 0 0 13.0000 #> 351 1 2 female 15.0000 0 2 39.0000 #> 352 0 2 male 60.0000 1 1 39.0000 #> 353 1 2 female 40.0000 1 1 39.0000 #> 354 1 2 female 20.0000 1 0 26.0000 #> 356 1 2 female 36.0000 0 0 13.0000 #> 357 0 2 male 25.0000 0 0 13.0000 #> 358 0 2 male 42.0000 0 0 13.0000 #> 359 1 2 female 42.0000 0 0 13.0000 #> 360 1 2 male 0.8333 0 2 29.0000 #> 361 1 2 male 26.0000 1 1 29.0000 #> 362 1 2 female 22.0000 1 1 29.0000 #> 363 1 2 female 35.0000 0 0 21.0000 #> 364 0 2 male NA 0 0 0.0000 #> 365 0 2 male 19.0000 0 0 13.0000 #> 366 0 2 female 44.0000 1 0 26.0000 #> 367 0 2 male 54.0000 1 0 26.0000 #> 368 0 2 male 52.0000 0 0 13.5000 #> 369 0 2 male 37.0000 1 0 26.0000 #> 370 0 2 female 29.0000 1 0 26.0000 #> 371 1 2 female 25.0000 1 1 30.0000 #> 372 1 2 female 45.0000 0 2 30.0000 #> 373 0 2 male 29.0000 1 0 26.0000 #> 374 1 2 female 28.0000 1 0 26.0000 #> 375 0 2 male 29.0000 0 0 10.5000 #> 376 0 2 male 28.0000 0 0 13.0000 #> 378 1 2 female 8.0000 0 2 26.2500 #> 379 0 2 male 31.0000 1 1 26.2500 #> 381 1 2 female 22.0000 0 0 10.5000 #> 382 0 2 female 30.0000 0 0 13.0000 #> 383 0 2 female NA 0 0 21.0000 #> 384 0 2 male 21.0000 0 0 11.5000 #> 385 0 2 male NA 0 0 0.0000 #> 386 1 2 male 8.0000 1 1 36.7500 #> 387 0 2 male 18.0000 0 0 73.5000 #> 388 1 2 female 48.0000 0 2 36.7500 #> 389 1 2 female 28.0000 0 0 13.0000 #> 390 0 2 male 32.0000 0 0 13.0000 #> 391 0 2 male 17.0000 0 0 73.5000 #> 392 0 2 male 29.0000 1 0 27.7208 #> 393 1 2 female 24.0000 1 0 27.7208 #> 394 0 2 male 25.0000 0 0 31.5000 #> 395 0 2 male 18.0000 0 0 73.5000 #> 396 1 2 female 18.0000 0 1 23.0000 #> 397 1 2 female 34.0000 0 1 23.0000 #> 398 0 2 male 54.0000 0 0 26.0000 #> 399 1 2 male 8.0000 0 2 32.5000 #> 400 0 2 male 42.0000 1 1 32.5000 #> 401 1 2 female 34.0000 1 1 32.5000 #> 402 1 2 female 27.0000 1 0 13.8583 #> 403 1 2 female 30.0000 1 0 13.8583 #> 404 0 2 male 23.0000 0 0 13.0000 #> 405 0 2 male 21.0000 0 0 13.0000 #> 406 0 2 male 18.0000 0 0 13.0000 #> 407 0 2 male 40.0000 1 0 26.0000 #> 408 1 2 female 29.0000 1 0 26.0000 #> 409 0 2 male 18.0000 0 0 10.5000 #> 410 0 2 male 36.0000 0 0 13.0000 #> 411 0 2 male NA 0 0 0.0000 #> 412 0 2 female 38.0000 0 0 13.0000 #> 413 0 2 male 35.0000 0 0 26.0000 #> 414 0 2 male 38.0000 1 0 21.0000 #> 415 0 2 male 34.0000 1 0 21.0000 #> 416 1 2 female 34.0000 0 0 13.0000 #> 417 0 2 male 16.0000 0 0 26.0000 #> 418 0 2 male 26.0000 0 0 10.5000 #> 419 0 2 male 47.0000 0 0 10.5000 #> 420 0 2 male 21.0000 1 0 11.5000 #> 421 0 2 male 21.0000 1 0 11.5000 #> 422 0 2 male 24.0000 0 0 13.5000 #> 423 0 2 male 24.0000 0 0 13.0000 #> 424 0 2 male 34.0000 0 0 13.0000 #> 425 0 2 male 30.0000 0 0 13.0000 #> 427 0 2 male 30.0000 0 0 13.0000 #> 428 1 2 male 0.6667 1 1 14.5000 #> 429 1 2 female 24.0000 0 2 14.5000 #> 430 0 2 male 44.0000 0 0 13.0000 #> 431 1 2 female 6.0000 0 1 33.0000 #> 432 0 2 male 28.0000 0 1 33.0000 #> 433 1 2 male 62.0000 0 0 10.5000 #> 434 0 2 male 30.0000 0 0 10.5000 #> 435 1 2 female 7.0000 0 2 26.2500 #> 436 0 2 male 43.0000 1 1 26.2500 #> 438 1 2 female 24.0000 1 2 65.0000 #> 439 1 2 female 24.0000 1 2 65.0000 #> 440 0 2 male 49.0000 1 2 65.0000 #> 441 1 2 female 48.0000 1 2 65.0000 #> 442 1 2 female 55.0000 0 0 16.0000 #> 443 0 2 male 24.0000 2 0 73.5000 #> 444 0 2 male 32.0000 2 0 73.5000 #> 445 0 2 male 21.0000 2 0 73.5000 #> 446 0 2 female 18.0000 1 1 13.0000 #> 447 1 2 female 20.0000 2 1 23.0000 #> 448 0 2 male 23.0000 2 1 11.5000 #> 449 0 2 male 36.0000 0 0 13.0000 #> 450 1 2 female 54.0000 1 3 23.0000 #> 451 0 2 male 50.0000 0 0 13.0000 #> 452 0 2 male 44.0000 1 0 26.0000 #> 453 1 2 female 29.0000 1 0 26.0000 #> 454 0 2 male 21.0000 0 0 73.5000 #> 455 1 2 male 42.0000 0 0 13.0000 #> 456 0 2 male 63.0000 1 0 26.0000 #> 457 0 2 female 60.0000 1 0 26.0000 #> 458 0 2 male 33.0000 0 0 12.2750 #> 459 1 2 female 17.0000 0 0 10.5000 #> 460 0 2 male 42.0000 1 0 27.0000 #> 461 1 2 female 24.0000 2 1 27.0000 #> 462 0 2 male 47.0000 0 0 15.0000 #> 463 0 2 male 24.0000 2 0 31.5000 #> 464 0 2 male 22.0000 2 0 31.5000 #> 465 0 2 male 32.0000 0 0 10.5000 #> 466 1 2 female 23.0000 0 0 13.7917 #> 467 0 2 male 34.0000 1 0 26.0000 #> 468 1 2 female 24.0000 1 0 26.0000 #> 469 0 2 female 22.0000 0 0 21.0000 #> 470 1 2 female NA 0 0 12.3500 #> 471 0 2 male 35.0000 0 0 12.3500 #> 472 1 2 female 45.0000 0 0 13.5000 #> 473 0 2 male 57.0000 0 0 12.3500 #> 474 0 2 male NA 0 0 0.0000 #> 475 0 2 male 31.0000 0 0 10.5000 #> 476 0 2 female 26.0000 1 1 26.0000 #> 477 0 2 male 30.0000 1 1 26.0000 #> 478 0 2 male NA 0 0 10.7083 #> 479 1 2 female 1.0000 1 2 41.5792 #> 480 1 2 female 3.0000 1 2 41.5792 #> 481 0 2 male 25.0000 1 2 41.5792 #> 482 1 2 female 22.0000 1 2 41.5792 #> 483 1 2 female 17.0000 0 0 12.0000 #> 484 1 2 female NA 0 0 33.0000 #> 485 1 2 female 34.0000 0 0 10.5000 #> 486 0 2 male 36.0000 0 0 12.8750 #> 487 0 2 male 24.0000 0 0 10.5000 #> 488 0 2 male 61.0000 0 0 12.3500 #> 489 0 2 male 50.0000 1 0 26.0000 #> 490 1 2 female 42.0000 1 0 26.0000 #> 492 0 2 male NA 0 0 15.0458 #> 493 1 2 male 1.0000 0 2 37.0042 #> 494 0 2 male 31.0000 1 1 37.0042 #> 495 1 2 female 24.0000 1 1 37.0042 #> 496 0 2 male NA 0 0 15.5792 #> 497 0 2 male 30.0000 0 0 13.0000 #> 500 0 2 male 30.0000 0 0 13.0000 #> 501 0 2 male 46.0000 0 0 26.0000 #> 502 1 2 female 13.0000 0 1 19.5000 #> 503 1 2 female 41.0000 0 1 19.5000 #> 504 1 2 male 19.0000 0 0 10.5000 #> 506 0 2 male 48.0000 0 0 13.0000 #> 508 0 2 male 27.0000 0 0 13.0000 #> 509 0 2 male 54.0000 0 0 14.0000 #> 510 0 2 male 39.0000 0 0 26.0000 #> 511 0 2 male 16.0000 0 0 10.5000 #> 512 0 2 male 62.0000 0 0 9.6875 #> 513 0 2 male 32.5000 1 0 30.0708 #> 514 1 2 female 14.0000 1 0 30.0708 #> 515 1 2 male 2.0000 1 1 26.0000 #> 516 1 2 male 3.0000 1 1 26.0000 #> 518 0 2 male 26.0000 0 0 13.0000 #> 519 0 2 male 19.0000 1 1 36.7500 #> 520 0 2 male 28.0000 0 0 13.5000 #> 521 1 2 male 20.0000 0 0 13.8625 #> 522 1 2 female 29.0000 0 0 10.5000 #> 523 0 2 male 39.0000 0 0 13.0000 #> 524 1 2 male 22.0000 0 0 10.5000 #> 525 1 2 male NA 0 0 13.8625 #> 526 0 2 male 23.0000 0 0 10.5000 #> 527 1 2 male 29.0000 0 0 13.8583 #> 528 0 2 male 28.0000 0 0 10.5000 #> 529 0 2 male NA 0 0 0.0000 #> 531 0 2 male 19.0000 0 0 10.5000 #> 532 0 2 male NA 0 0 15.0500 #> 533 0 2 male 41.0000 0 0 13.0000 #> 534 1 2 female 21.0000 0 1 21.0000 #> 535 1 2 female 19.0000 0 0 26.0000 #> 536 0 2 male 43.0000 0 1 21.0000 #> 537 1 2 female 32.0000 0 0 13.0000 #> 538 0 2 male 34.0000 0 0 13.0000 #> 539 1 2 male 30.0000 0 0 12.7375 #> 540 0 2 male 27.0000 0 0 15.0333 #> 542 1 2 female 8.0000 1 1 26.0000 #> 543 1 2 female 33.0000 0 2 26.0000 #> 546 1 2 female 30.0000 3 0 21.0000 #> 547 1 2 female 28.0000 0 0 13.0000 #> 548 0 2 male 23.0000 0 0 15.0458 #> 550 1 2 male 3.0000 1 1 18.7500 #> 551 1 2 female 24.0000 2 3 18.7500 #> 552 1 2 female 50.0000 0 0 10.5000 #> 554 1 2 female 21.0000 0 0 10.5000 #> 555 0 2 male 26.0000 0 0 13.0000 #> 556 0 2 male 25.0000 0 0 13.0000 #> 557 0 2 male 27.0000 0 0 26.0000 #> 558 1 2 female 25.0000 0 1 26.0000 #> 559 1 2 female 18.0000 0 2 13.0000 #> 560 1 2 female 20.0000 0 0 36.7500 #> 561 1 2 female 30.0000 0 0 13.0000 #> 562 0 2 male 59.0000 0 0 13.5000 #> 563 1 2 female 30.0000 0 0 12.3500 #> 564 0 2 male 35.0000 0 0 10.5000 #> 565 1 2 female 40.0000 0 0 13.0000 #> 566 0 2 male 25.0000 0 0 13.0000 #> 567 0 2 male 41.0000 0 0 15.0458 #> 568 0 2 male 25.0000 0 0 10.5000 #> 569 0 2 male 18.5000 0 0 13.0000 #> 570 0 2 male 14.0000 0 0 65.0000 #> 571 1 2 female 50.0000 0 0 10.5000 #> 572 0 2 male 23.0000 0 0 13.0000 #> 573 1 2 female 28.0000 0 0 12.6500 #> 574 1 2 female 27.0000 0 0 10.5000 #> 575 0 2 male 29.0000 1 0 21.0000 #> 576 0 2 female 27.0000 1 0 21.0000 #> 577 0 2 male 40.0000 0 0 13.0000 #> 578 1 2 female 31.0000 0 0 21.0000 #> 579 0 2 male 30.0000 1 0 21.0000 #> 580 0 2 male 23.0000 1 0 10.5000 #> 581 1 2 female 31.0000 0 0 21.0000 #> 583 1 2 female 12.0000 0 0 15.7500 #> 584 1 2 female 40.0000 0 0 15.7500 #> 587 1 2 female 29.0000 1 0 26.0000 #> 588 1 2 male 2.0000 1 1 23.0000 #> 590 1 2 female 29.0000 0 2 23.0000 #> 591 1 2 female 0.9167 1 2 27.7500 #> 592 1 2 female 5.0000 1 2 27.7500 #> 594 1 2 female 33.0000 1 2 27.7500 #> 595 0 2 male 66.0000 0 0 10.5000 #> 596 0 2 male NA 0 0 12.8750 #> 597 1 2 male 31.0000 0 0 13.0000 #> 598 1 2 male NA 0 0 13.0000 #> 599 1 2 female 26.0000 0 0 13.5000 #> 600 0 2 female 24.0000 0 0 13.0000 #> 601 0 3 male 42.0000 0 0 7.5500 #> 602 0 3 male 13.0000 0 2 20.2500 #> 603 0 3 male 16.0000 1 1 20.2500 #> 604 1 3 female 35.0000 1 1 20.2500 #> 605 1 3 female 16.0000 0 0 7.6500 #> 606 1 3 male 25.0000 0 0 7.6500 #> 607 1 3 male 20.0000 0 0 7.9250 #> 608 1 3 female 18.0000 0 0 7.2292 #> 609 0 3 male 30.0000 0 0 7.2500 #> 610 0 3 male 26.0000 0 0 8.0500 #> 611 0 3 female 40.0000 1 0 9.4750 #> 612 1 3 male 0.8333 0 1 9.3500 #> 613 1 3 female 18.0000 0 1 9.3500 #> 614 1 3 male 26.0000 0 0 18.7875 #> 615 0 3 male 26.0000 0 0 7.8875 #> 616 0 3 male 20.0000 0 0 7.9250 #> 617 0 3 male 24.0000 0 0 7.0500 #> 619 0 3 male 35.0000 0 0 8.0500 #> 620 0 3 male 18.0000 0 0 8.3000 #> 621 0 3 male 32.0000 0 0 22.5250 #> 622 1 3 female 19.0000 1 0 7.8542 #> 623 0 3 male 4.0000 4 2 31.2750 #> 624 0 3 female 6.0000 4 2 31.2750 #> 625 0 3 female 2.0000 4 2 31.2750 #> 627 0 3 female 38.0000 4 2 7.7750 #> 628 0 3 female 9.0000 4 2 31.2750 #> 629 0 3 female 11.0000 4 2 31.2750 #> 630 0 3 male 39.0000 1 5 31.2750 #> 631 1 3 male 27.0000 0 0 7.7958 #> 632 0 3 male 26.0000 0 0 7.7750 #> 634 0 3 male 20.0000 0 0 7.8542 #> 635 0 3 male 26.0000 0 0 7.8958 #> 636 0 3 male 25.0000 1 0 17.8000 #> 637 0 3 female 18.0000 1 0 17.8000 #> 638 0 3 male 24.0000 0 0 7.7750 #> 639 0 3 male 35.0000 0 0 7.0500 #> 640 0 3 male 5.0000 4 2 31.3875 #> 642 1 3 male 3.0000 4 2 31.3875 #> 643 0 3 male 13.0000 4 2 31.3875 #> 644 1 3 female 5.0000 4 2 31.3875 #> 645 0 3 male 40.0000 1 5 31.3875 #> 646 1 3 male 23.0000 0 0 7.7958 #> 647 1 3 female 38.0000 1 5 31.3875 #> 648 1 3 female 45.0000 0 0 7.2250 #> 649 0 3 male 21.0000 0 0 7.2250 #> 650 0 3 male 23.0000 0 0 7.0500 #> 651 0 3 female 17.0000 0 0 14.4583 #> 652 0 3 male 30.0000 0 0 7.2250 #> 653 0 3 male 23.0000 0 0 7.8542 #> 654 1 3 female 13.0000 0 0 7.2292 #> 655 0 3 male 20.0000 0 0 7.2250 #> 656 0 3 male 32.0000 1 0 15.8500 #> 658 1 3 female 0.7500 2 1 19.2583 #> 659 1 3 female 0.7500 2 1 19.2583 #> 660 1 3 female 5.0000 2 1 19.2583 #> 661 1 3 female 24.0000 0 3 19.2583 #> 662 1 3 female 18.0000 0 0 8.0500 #> 663 0 3 male 40.0000 0 0 7.2250 #> 664 0 3 male 26.0000 0 0 7.8958 #> 665 1 3 male 20.0000 0 0 7.2292 #> 666 0 3 female 18.0000 0 1 14.4542 #> 667 0 3 female 45.0000 0 1 14.4542 #> 668 0 3 female 27.0000 0 0 7.8792 #> 670 0 3 male 19.0000 0 0 8.0500 #> 671 0 3 male 26.0000 0 0 7.7750 #> 672 0 3 male 22.0000 0 0 9.3500 #> 673 0 3 male NA 0 0 7.2292 #> 674 0 3 male 20.0000 0 0 4.0125 #> 676 0 3 male 21.0000 0 0 7.7750 #> 677 0 3 male 18.0000 0 0 7.7500 #> 678 0 3 male 26.0000 0 0 7.8958 #> 679 0 3 male 6.0000 1 1 15.2458 #> 680 0 3 female 9.0000 1 1 15.2458 #> 681 0 3 male NA 0 0 7.2250 #> 682 0 3 female NA 0 2 15.2458 #> 683 0 3 female NA 0 2 7.7500 #> 684 0 3 male 40.0000 1 1 15.5000 #> 685 0 3 female 32.0000 1 1 15.5000 #> 686 0 3 male 21.0000 0 0 16.1000 #> 687 1 3 female 22.0000 0 0 7.7250 #> 688 0 3 female 20.0000 0 0 7.8542 #> 689 0 3 male 29.0000 1 0 7.0458 #> 690 0 3 male 22.0000 1 0 7.2500 #> 691 0 3 male 22.0000 0 0 7.7958 #> 692 0 3 male 35.0000 0 0 8.0500 #> 693 0 3 female 18.5000 0 0 7.2833 #> 694 1 3 male 21.0000 0 0 7.8208 #> 695 0 3 male 19.0000 0 0 6.7500 #> 696 0 3 female 18.0000 0 0 7.8792 #> 697 0 3 female 21.0000 0 0 8.6625 #> 698 0 3 female 30.0000 0 0 8.6625 #> 700 0 3 male 38.0000 0 0 8.6625 #> 703 0 3 female 21.0000 0 0 7.7500 #> 704 0 3 male 21.0000 0 0 7.7500 #> 705 0 3 male 21.0000 0 0 8.0500 #> 706 0 3 male NA 1 0 14.4583 #> 707 0 3 female NA 1 0 14.4583 #> 708 0 3 male 28.0000 0 0 7.7958 #> 709 0 3 male 24.0000 0 0 7.8542 #> 710 1 3 female 16.0000 0 0 7.7500 #> 711 0 3 female 37.0000 0 0 7.7500 #> 712 0 3 male 28.0000 0 0 7.2500 #> 713 0 3 male 24.0000 0 0 8.0500 #> 714 0 3 male 21.0000 0 0 7.7333 #> 715 1 3 male 32.0000 0 0 56.4958 #> 717 0 3 male 26.0000 1 0 14.4542 #> 718 0 3 male 18.0000 1 0 14.4542 #> 719 0 3 male 20.0000 0 0 7.0500 #> 720 1 3 male 18.0000 0 0 8.0500 #> 721 0 3 male 24.0000 0 0 7.2500 #> 722 0 3 male 36.0000 0 0 7.4958 #> 723 0 3 male 24.0000 0 0 7.4958 #> 724 0 3 male 31.0000 0 0 7.7333 #> 725 0 3 male 31.0000 0 0 7.7500 #> 726 1 3 female 22.0000 0 0 7.7500 #> 727 0 3 female 30.0000 0 0 7.6292 #> 729 0 3 male 43.0000 0 0 8.0500 #> 730 0 3 male 35.0000 0 0 7.8958 #> 732 0 3 male 19.0000 0 0 7.8958 #> 733 0 3 male 30.0000 0 0 8.0500 #> 734 1 3 male 9.0000 1 1 15.9000 #> 735 1 3 male 3.0000 1 1 15.9000 #> 739 1 3 female 17.0000 0 1 16.1000 #> 740 0 3 male 44.0000 0 1 16.1000 #> 742 0 3 male 22.5000 0 0 7.2250 #> 743 1 3 male 45.0000 0 0 8.0500 #> 744 0 3 female 22.0000 0 0 10.5167 #> 745 0 3 male 19.0000 0 0 10.1708 #> 746 1 3 female 30.0000 0 0 6.9500 #> 747 1 3 male 29.0000 0 0 7.7500 #> 748 0 3 male 0.3333 0 2 14.4000 #> 749 0 3 male 34.0000 1 1 14.4000 #> 750 0 3 female 28.0000 1 1 14.4000 #> 751 0 3 male 27.0000 0 0 7.8958 #> 752 0 3 male 25.0000 0 0 7.8958 #> 753 0 3 male 24.0000 2 0 24.1500 #> 754 0 3 male 22.0000 0 0 8.0500 #> 755 0 3 male 21.0000 2 0 24.1500 #> 757 0 3 male NA 1 0 16.1000 #> 758 1 3 female NA 1 0 16.1000 #> 759 1 3 male 36.5000 1 0 17.4000 #> 760 1 3 female 36.0000 1 0 17.4000 #> 761 1 3 male 30.0000 0 0 9.5000 #> 762 0 3 male 16.0000 0 0 9.5000 #> 763 1 3 male 1.0000 1 2 20.5750 #> 765 0 3 male 26.0000 1 2 20.5750 #> 766 1 3 female 33.0000 1 2 20.5750 #> 767 0 3 male 25.0000 0 0 7.8958 #> 768 0 3 male NA 0 0 7.8958 #> 769 0 3 male NA 0 0 7.8958 #> 770 0 3 male 22.0000 0 0 7.2500 #> 771 0 3 male 36.0000 0 0 7.2500 #> 772 1 3 female 19.0000 0 0 7.8792 #> 773 0 3 male 17.0000 0 0 7.8958 #> 774 0 3 male 42.0000 0 0 8.6625 #> 775 0 3 male 43.0000 0 0 7.8958 #> 776 0 3 male NA 0 0 7.2292 #> 777 0 3 male 32.0000 0 0 7.7500 #> 778 1 3 male 19.0000 0 0 8.0500 #> 779 1 3 female 30.0000 0 0 12.4750 #> 780 0 3 female 24.0000 0 0 7.7500 #> 781 1 3 female 23.0000 0 0 8.0500 #> 782 0 3 male 33.0000 0 0 7.8958 #> 783 0 3 male 65.0000 0 0 7.7500 #> 784 1 3 male 24.0000 0 0 7.5500 #> 785 0 3 male 23.0000 1 0 13.9000 #> 786 1 3 female 22.0000 1 0 13.9000 #> 787 0 3 male 18.0000 0 0 7.7750 #> 788 0 3 male 16.0000 0 0 7.7750 #> 789 0 3 male 45.0000 0 0 6.9750 #> 790 0 3 male NA 0 0 7.2250 #> 791 0 3 male 39.0000 0 2 7.2292 #> 792 0 3 male 17.0000 1 1 7.2292 #> 793 0 3 male 15.0000 1 1 7.2292 #> 794 0 3 male 47.0000 0 0 7.2500 #> 796 0 3 male NA 0 0 7.2250 #> 797 0 3 male 40.5000 0 0 15.1000 #> 798 0 3 male 40.5000 0 0 7.7500 #> 799 1 3 male NA 0 0 7.0500 #> 800 0 3 male 18.0000 0 0 7.7958 #> 801 0 3 female NA 0 0 7.7500 #> 802 0 3 male NA 0 0 7.7500 #> 804 0 3 male 26.0000 0 0 7.8792 #> 805 0 3 male NA 0 0 7.7500 #> 807 0 3 female 21.0000 2 2 34.3750 #> 809 0 3 male NA 0 0 8.0500 #> 810 0 3 male 18.0000 2 2 34.3750 #> 811 0 3 male 16.0000 1 3 34.3750 #> 812 0 3 female 48.0000 1 3 34.3750 #> 813 0 3 male NA 0 0 7.7500 #> 814 0 3 male NA 0 0 7.2500 #> 815 0 3 male 25.0000 0 0 7.7417 #> 816 0 3 male NA 0 0 14.5000 #> 817 0 3 male NA 0 0 7.8958 #> 818 0 3 male 22.0000 0 0 8.0500 #> 819 1 3 female 16.0000 0 0 7.7333 #> 820 1 3 female NA 0 0 7.7500 #> 821 1 3 male 9.0000 0 2 20.5250 #> 822 0 3 male 33.0000 1 1 20.5250 #> 823 0 3 male 41.0000 0 0 7.8500 #> 824 1 3 female 31.0000 1 1 20.5250 #> 825 0 3 male 38.0000 0 0 7.0500 #> 826 0 3 male 9.0000 5 2 46.9000 #> 827 0 3 male 1.0000 5 2 46.9000 #> 828 0 3 male 11.0000 5 2 46.9000 #> 829 0 3 female 10.0000 5 2 46.9000 #> 830 0 3 female 16.0000 5 2 46.9000 #> 831 0 3 male 14.0000 5 2 46.9000 #> 832 0 3 male 40.0000 1 6 46.9000 #> 833 0 3 female 43.0000 1 6 46.9000 #> 834 0 3 male 51.0000 0 0 8.0500 #> 835 0 3 male 32.0000 0 0 8.3625 #> 837 0 3 male 20.0000 0 0 9.8458 #> 838 0 3 male 37.0000 2 0 7.9250 #> 839 0 3 male 28.0000 2 0 7.9250 #> 840 0 3 male 19.0000 0 0 7.7750 #> 841 0 3 female 24.0000 0 0 8.8500 #> 842 0 3 female 17.0000 0 0 7.7333 #> 843 0 3 male NA 1 0 19.9667 #> 844 0 3 male NA 1 0 19.9667 #> 845 0 3 male 28.0000 1 0 15.8500 #> 846 1 3 female 24.0000 1 0 15.8500 #> 847 0 3 male 20.0000 0 0 9.5000 #> 849 0 3 male 41.0000 2 0 14.1083 #> 850 0 3 male 26.0000 1 0 7.8542 #> 851 0 3 male 21.0000 0 0 7.8542 #> 852 1 3 female 45.0000 1 0 14.1083 #> 853 0 3 female NA 0 0 7.5500 #> 854 0 3 male 25.0000 0 0 7.2500 #> 855 0 3 male NA 0 0 6.8583 #> 856 0 3 male 11.0000 0 0 18.7875 #> 857 1 3 female NA 0 0 7.7500 #> 859 1 3 male NA 0 0 56.4958 #> 860 0 3 female 18.0000 0 0 6.7500 #> 861 1 3 female 26.0000 0 0 7.9250 #> 862 0 3 female 23.0000 0 0 7.9250 #> 863 1 3 female 22.0000 0 0 8.9625 #> 864 0 3 male 28.0000 0 0 7.8958 #> 865 0 3 female 28.0000 0 0 7.7750 #> 866 0 3 female NA 0 0 7.7500 #> 868 1 3 female 22.0000 1 1 12.2875 #> 869 0 3 male 43.0000 0 0 6.4500 #> 870 0 3 male 28.0000 0 0 22.5250 #> 871 1 3 female 27.0000 0 0 7.9250 #> 872 0 3 male NA 0 0 7.7500 #> 873 1 3 female NA 0 0 8.0500 #> 874 0 3 male 42.0000 0 0 7.6500 #> 875 1 3 male NA 0 0 7.8875 #> 876 0 3 male 30.0000 0 0 7.2292 #> 877 0 3 male NA 0 0 7.8958 #> 878 0 3 female 27.0000 1 0 7.9250 #> 879 0 3 female 25.0000 1 0 7.9250 #> 880 0 3 male NA 0 0 7.8958 #> 882 1 3 male 21.0000 0 0 7.7958 #> 883 0 3 male NA 0 0 7.0500 #> 884 0 3 male 20.0000 0 0 7.8542 #> 886 0 3 male 17.0000 1 0 7.0542 #> 887 1 3 female NA 0 0 7.7500 #> 888 1 3 male NA 0 0 8.1125 #> 889 0 3 male 34.0000 0 0 6.4958 #> 890 1 3 male 26.0000 0 0 7.7750 #> 891 0 3 male 22.0000 0 0 7.7958 #> 892 0 3 male 33.0000 0 0 8.6542 #> 894 0 3 male 29.0000 0 0 7.8542 #> 895 1 3 male 4.0000 1 1 11.1333 #> 896 1 3 female 1.0000 1 1 11.1333 #> 897 0 3 male 49.0000 0 0 0.0000 #> 898 0 3 male 33.0000 0 0 7.7750 #> 899 0 3 male 19.0000 0 0 0.0000 #> 900 1 3 female 27.0000 0 2 11.1333 #> 901 0 3 male NA 1 2 23.4500 #> 902 0 3 female NA 1 2 23.4500 #> 903 0 3 male NA 1 2 23.4500 #> 904 0 3 female NA 1 2 23.4500 #> 906 1 3 male 32.0000 0 0 7.8542 #> 907 0 3 male 27.0000 0 0 7.8542 #> 908 0 3 female 20.0000 1 0 9.8250 #> 909 0 3 female 21.0000 1 0 9.8250 #> 910 1 3 male 32.0000 0 0 7.9250 #> 911 0 3 male 17.0000 0 0 7.1250 #> 912 0 3 male 21.0000 0 0 8.4333 #> 913 0 3 male 30.0000 0 0 7.8958 #> 914 1 3 male 21.0000 0 0 7.7958 #> 916 0 3 male 22.0000 0 0 7.5208 #> 917 1 3 female 4.0000 0 1 13.4167 #> 918 1 3 male 39.0000 0 1 13.4167 #> 919 0 3 male NA 0 0 7.2292 #> 920 0 3 male 18.5000 0 0 7.2292 #> 922 0 3 male NA 0 0 7.2500 #> 923 1 3 female NA 0 0 7.7500 #> 924 1 3 female NA 0 0 7.7500 #> 926 0 3 male 44.0000 0 0 8.0500 #> 927 1 3 male NA 0 0 7.7500 #> 928 0 3 male NA 1 0 14.4542 #> 929 0 3 female NA 1 0 14.4542 #> 931 0 3 male NA 1 0 7.7500 #> 932 0 3 male NA 0 0 7.7375 #> 933 0 3 female 22.0000 2 0 8.6625 #> 934 0 3 male 26.0000 2 0 8.6625 #> 935 1 3 female 4.0000 0 2 22.0250 #> 936 1 3 male 29.0000 3 1 22.0250 #> 937 1 3 female 26.0000 1 1 22.0250 #> 938 0 3 female 1.0000 1 1 12.1833 #> 939 0 3 male 18.0000 1 1 7.8542 #> 940 0 3 female 36.0000 0 2 12.1833 #> 942 1 3 male 25.0000 0 0 7.2292 #> 943 0 3 male NA 0 0 7.2250 #> 944 0 3 female 37.0000 0 0 9.5875 #> 945 0 3 male NA 0 0 7.8958 #> 946 1 3 male NA 0 0 56.4958 #> 949 0 3 male NA 0 0 7.7500 #> 950 1 3 male 26.0000 0 0 56.4958 #> 951 0 3 male 29.0000 0 0 9.4833 #> 952 0 3 male 29.0000 0 0 7.7750 #> 953 0 3 male 22.0000 0 0 7.7750 #> 954 1 3 male 22.0000 0 0 7.2250 #> 955 0 3 male NA 3 1 25.4667 #> 956 0 3 female NA 3 1 25.4667 #> 957 0 3 female NA 3 1 25.4667 #> 958 0 3 female NA 3 1 25.4667 #> 960 0 3 male 32.0000 0 0 7.9250 #> 961 0 3 male 34.5000 0 0 6.4375 #> 962 0 3 female NA 1 0 15.5000 #> 963 0 3 male NA 1 0 15.5000 #> 964 0 3 male 36.0000 0 0 0.0000 #> 965 0 3 male 39.0000 0 0 24.1500 #> 966 0 3 male 24.0000 0 0 9.5000 #> 968 0 3 female 45.0000 0 0 7.7500 #> 969 0 3 male 36.0000 1 0 15.5500 #> 970 0 3 female 30.0000 1 0 15.5500 #> 972 0 3 male NA 0 0 7.8792 #> 973 0 3 male 28.0000 0 0 56.4958 #> 974 0 3 male NA 0 0 7.5500 #> 976 0 3 female 26.0000 1 0 16.1000 #> 977 0 3 male NA 0 0 7.8792 #> 978 0 3 male 20.5000 0 0 7.2500 #> 979 1 3 male 27.0000 0 0 8.6625 #> 980 0 3 male 51.0000 0 0 7.0542 #> 981 1 3 female 23.0000 0 0 7.8542 #> 982 1 3 male 32.0000 0 0 7.5792 #> 983 0 3 male NA 0 0 7.8958 #> 984 0 3 male NA 0 0 7.5500 #> 985 1 3 female NA 0 0 7.7500 #> 986 1 3 male 24.0000 0 0 7.1417 #> 987 0 3 male 22.0000 0 0 7.1250 #> 988 0 3 female NA 0 0 7.8792 #> 989 0 3 male NA 0 0 7.7500 #> 990 0 3 male NA 0 0 8.0500 #> 991 0 3 male 29.0000 0 0 7.9250 #> 993 0 3 female 30.5000 0 0 7.7500 #> 994 1 3 female NA 0 0 7.7375 #> 995 0 3 male NA 0 0 7.2292 #> 996 0 3 male 35.0000 0 0 7.8958 #> 997 0 3 male 33.0000 0 0 7.8958 #> 998 1 3 female NA 0 0 7.2250 #> 1001 1 3 male NA 0 0 7.7500 #> 1002 1 3 female NA 2 0 23.2500 #> 1003 1 3 female NA 2 0 23.2500 #> 1004 1 3 male NA 2 0 23.2500 #> 1005 1 3 female NA 0 0 7.7875 #> 1006 0 3 male NA 0 0 15.5000 #> 1007 1 3 female NA 0 0 7.8792 #> 1008 1 3 female 15.0000 0 0 8.0292 #> 1009 0 3 female 35.0000 0 0 7.7500 #> 1010 0 3 male NA 0 0 7.7500 #> 1011 0 3 male 24.0000 1 0 16.1000 #> 1012 0 3 female 19.0000 1 0 16.1000 #> 1015 0 3 female NA 0 0 8.0500 #> 1016 0 3 male 55.5000 0 0 8.0500 #> 1017 0 3 male NA 0 0 7.7500 #> 1018 1 3 male 21.0000 0 0 7.7750 #> 1020 0 3 male 24.0000 0 0 7.8958 #> 1021 0 3 male 21.0000 0 0 7.8958 #> 1022 0 3 male 28.0000 0 0 7.8958 #> 1023 0 3 male NA 0 0 7.8958 #> 1024 1 3 female NA 0 0 7.8792 #> 1025 0 3 male 25.0000 0 0 7.6500 #> 1026 1 3 male 6.0000 0 1 12.4750 #> 1027 1 3 female 27.0000 0 1 12.4750 #> 1028 0 3 male NA 0 0 8.0500 #> 1030 0 3 male NA 1 0 24.1500 #> 1032 0 3 male 34.0000 0 0 8.0500 #> 1033 0 3 male NA 0 0 7.7500 #> 1034 1 3 male NA 0 0 7.7750 #> 1035 1 3 male NA 1 1 15.2458 #> 1036 1 3 male NA 1 1 15.2458 #> 1037 1 3 female NA 0 2 15.2458 #> 1038 1 3 female NA 0 0 7.2292 #> 1039 0 3 male NA 0 0 8.0500 #> 1040 1 3 female NA 0 0 7.7333 #> 1041 1 3 female 24.0000 0 0 7.7500 #> 1042 0 3 male NA 0 0 8.0500 #> 1043 1 3 female NA 1 0 15.5000 #> 1044 1 3 female NA 1 0 15.5000 #> 1046 0 3 male 18.0000 0 0 7.7500 #> 1047 0 3 male 22.0000 0 0 7.8958 #> 1048 1 3 female 15.0000 0 0 7.2250 #> 1049 1 3 female 1.0000 0 2 15.7417 #> 1050 1 3 male 20.0000 1 1 15.7417 #> 1051 1 3 female 19.0000 1 1 15.7417 #> 1052 0 3 male 33.0000 0 0 8.0500 #> 1053 0 3 male NA 0 0 7.8958 #> 1054 0 3 male NA 0 0 7.2292 #> 1055 0 3 female NA 0 0 7.7500 #> 1056 0 3 male NA 0 0 7.8958 #> 1057 1 3 male 12.0000 1 0 11.2417 #> 1058 1 3 female 14.0000 1 0 11.2417 #> 1059 0 3 female 29.0000 0 0 7.9250 #> 1060 0 3 male 28.0000 0 0 8.0500 #> 1061 1 3 female 18.0000 0 0 7.7750 #> 1062 1 3 female 26.0000 0 0 7.8542 #> 1063 0 3 male 21.0000 0 0 7.8542 #> 1064 0 3 male 41.0000 0 0 7.1250 #> 1065 1 3 male 39.0000 0 0 7.9250 #> 1066 0 3 male 21.0000 0 0 7.8000 #> 1067 0 3 male 28.5000 0 0 7.2292 #> 1068 1 3 female 22.0000 0 0 7.7500 #> 1069 0 3 male 61.0000 0 0 6.2375 #> 1070 0 3 male NA 1 0 15.5000 #> 1071 0 3 male NA 0 0 7.8292 #> 1072 1 3 female NA 1 0 15.5000 #> 1073 0 3 male NA 0 0 7.7333 #> 1074 0 3 male NA 0 0 7.7500 #> 1075 0 3 male NA 0 0 7.7500 #> 1076 0 3 male 23.0000 0 0 9.2250 #> 1077 0 3 female NA 0 0 7.7500 #> 1078 1 3 female NA 0 0 7.7500 #> 1079 1 3 female NA 0 0 7.8792 #> 1080 1 3 female 22.0000 0 0 7.7750 #> 1081 1 3 male NA 0 0 7.7500 #> 1082 1 3 female NA 0 0 7.8292 #> 1083 1 3 male 9.0000 0 1 3.1708 #> 1084 0 3 male 28.0000 0 0 22.5250 #> 1085 0 3 male 42.0000 0 1 8.4042 #> 1086 0 3 male NA 0 0 7.3125 #> 1087 0 3 female 31.0000 0 0 7.8542 #> 1088 0 3 male 28.0000 0 0 7.8542 #> 1089 1 3 male 32.0000 0 0 7.7750 #> 1092 0 3 female 20.0000 0 0 8.6625 #> 1093 0 3 male 20.0000 0 0 8.6625 #> 1094 0 3 male 16.0000 0 0 9.2167 #> 1095 1 3 female 31.0000 0 0 8.6833 #> 1096 0 3 female NA 0 0 7.6292 #> 1097 0 3 male 2.0000 3 1 21.0750 #> 1098 0 3 male 6.0000 3 1 21.0750 #> 1099 0 3 female 3.0000 3 1 21.0750 #> 1100 0 3 female 8.0000 3 1 21.0750 #> 1101 0 3 female 29.0000 0 4 21.0750 #> 1102 0 3 male 1.0000 4 1 39.6875 #> 1103 0 3 male 7.0000 4 1 39.6875 #> 1104 0 3 male 2.0000 4 1 39.6875 #> 1105 0 3 male 16.0000 4 1 39.6875 #> 1106 0 3 male 14.0000 4 1 39.6875 #> 1107 0 3 female 41.0000 0 5 39.6875 #> 1108 0 3 male 21.0000 0 0 8.6625 #> 1109 0 3 male 19.0000 0 0 14.5000 #> 1110 0 3 male NA 0 0 8.7125 #> 1111 0 3 male 32.0000 0 0 7.8958 #> 1112 0 3 male 0.7500 1 1 13.7750 #> 1113 0 3 female 3.0000 1 1 13.7750 #> 1114 0 3 female 26.0000 0 2 13.7750 #> 1116 0 3 male NA 0 0 7.7750 #> 1117 0 3 male NA 0 0 8.0500 #> 1118 0 3 male 21.0000 0 0 7.9250 #> 1119 0 3 male 25.0000 0 0 7.9250 #> 1120 0 3 male 22.0000 0 0 7.2500 #> 1121 1 3 male 25.0000 1 0 7.7750 #> 1122 1 3 male NA 1 1 22.3583 #> 1123 1 3 female NA 1 1 22.3583 #> 1124 1 3 female NA 0 2 22.3583 #> 1125 0 3 female NA 0 0 8.1375 #> 1126 0 3 male 24.0000 0 0 8.0500 #> 1128 0 3 male 19.0000 0 0 7.8958 #> 1129 0 3 male NA 0 0 7.8958 #> 1130 0 3 male 25.0000 1 0 7.7750 #> 1131 0 3 female 18.0000 0 0 7.7750 #> 1132 1 3 male 32.0000 0 0 8.0500 #> 1133 0 3 male NA 0 0 7.8958 #> 1134 0 3 male 17.0000 0 0 8.6625 #> 1135 0 3 male 24.0000 0 0 8.6625 #> 1136 0 3 male NA 0 0 7.8958 #> 1137 0 3 female NA 0 0 8.1125 #> 1138 0 3 male NA 0 0 7.2292 #> 1139 0 3 male NA 0 0 7.2500 #> 1142 0 3 male 10.0000 4 1 29.1250 #> 1143 0 3 male 4.0000 4 1 29.1250 #> 1144 0 3 male 7.0000 4 1 29.1250 #> 1145 0 3 male 2.0000 4 1 29.1250 #> 1146 0 3 male 8.0000 4 1 29.1250 #> 1147 0 3 female 39.0000 0 5 29.1250 #> 1148 0 3 female 22.0000 0 0 39.6875 #> 1149 0 3 male 35.0000 0 0 7.1250 #> 1150 1 3 female NA 0 0 7.7208 #> 1151 0 3 male NA 0 0 14.5000 #> 1152 0 3 female NA 0 0 14.5000 #> 1153 0 3 male 50.0000 1 0 14.5000 #> 1154 0 3 female 47.0000 1 0 14.5000 #> 1155 0 3 male NA 0 0 8.0500 #> 1156 0 3 male NA 0 0 7.7750 #> 1158 0 3 male 18.0000 1 1 20.2125 #> 1159 0 3 female 41.0000 0 2 20.2125 #> 1160 1 3 female NA 0 0 8.0500 #> 1161 0 3 male 50.0000 0 0 8.0500 #> 1162 0 3 male 16.0000 0 0 8.0500 #> 1163 1 3 male NA 0 0 7.7500 #> 1164 0 3 male NA 0 0 24.1500 #> 1165 0 3 male NA 0 0 7.2292 #> 1166 0 3 male 25.0000 0 0 7.2250 #> 1167 0 3 male NA 0 0 7.2250 #> 1169 0 3 male NA 0 0 7.5750 #> 1170 0 3 male 38.5000 0 0 7.2500 #> 1171 0 3 male NA 8 2 69.5500 #> 1172 0 3 male 14.5000 8 2 69.5500 #> 1173 0 3 female NA 8 2 69.5500 #> 1174 0 3 female NA 8 2 69.5500 #> 1175 0 3 female NA 8 2 69.5500 #> 1176 0 3 female NA 8 2 69.5500 #> 1177 0 3 male NA 8 2 69.5500 #> 1178 0 3 male NA 8 2 69.5500 #> 1179 0 3 male NA 8 2 69.5500 #> 1180 0 3 male NA 1 9 69.5500 #> 1181 0 3 female NA 1 9 69.5500 #> 1182 0 3 male 24.0000 0 0 9.3250 #> 1184 0 3 male 39.0000 0 0 7.9250 #> 1185 0 3 male NA 2 0 21.6792 #> 1186 0 3 male NA 2 0 21.6792 #> 1187 0 3 male NA 2 0 21.6792 #> 1188 1 3 female 1.0000 1 1 16.7000 #> 1189 1 3 female 24.0000 0 2 16.7000 #> 1190 1 3 female 4.0000 1 1 16.7000 #> 1191 1 3 male 25.0000 0 0 9.5000 #> 1192 0 3 male 20.0000 0 0 8.0500 #> 1193 0 3 male 24.5000 0 0 8.0500 #> 1194 0 3 male NA 0 0 7.7250 #> 1196 0 3 male NA 0 0 7.7500 #> 1197 1 3 male 29.0000 0 0 9.5000 #> 1198 0 3 male NA 0 0 15.1000 #> 1199 1 3 female NA 0 0 7.7792 #> 1200 0 3 male NA 0 0 8.0500 #> 1201 0 3 male NA 0 0 8.0500 #> 1202 0 3 male 22.0000 0 0 7.2292 #> 1203 0 3 male NA 0 0 8.0500 #> 1204 0 3 male 40.0000 0 0 7.8958 #> 1205 0 3 male 21.0000 0 0 7.9250 #> 1206 1 3 female 18.0000 0 0 7.4958 #> 1207 0 3 male 4.0000 3 2 27.9000 #> 1208 0 3 male 10.0000 3 2 27.9000 #> 1209 0 3 female 9.0000 3 2 27.9000 #> 1210 0 3 female 2.0000 3 2 27.9000 #> 1211 0 3 male 40.0000 1 4 27.9000 #> 1212 0 3 female 45.0000 1 4 27.9000 #> 1213 0 3 male NA 0 0 7.8958 #> 1214 0 3 male NA 0 0 8.0500 #> 1215 0 3 male NA 0 0 8.6625 #> 1216 0 3 male NA 0 0 7.7500 #> 1217 1 3 female NA 0 0 7.7333 #> 1218 0 3 male 19.0000 0 0 7.6500 #> 1219 0 3 male 30.0000 0 0 8.0500 #> 1220 0 3 male NA 0 0 8.0500 #> 1221 0 3 male 32.0000 0 0 8.0500 #> 1222 0 3 male NA 0 0 7.8958 #> 1223 0 3 male 33.0000 0 0 8.6625 #> 1224 1 3 female 23.0000 0 0 7.5500 #> 1225 0 3 male 21.0000 0 0 8.0500 #> 1226 0 3 male 60.5000 0 0 NA #> 1227 0 3 male 19.0000 0 0 7.8958 #> 1228 0 3 female 22.0000 0 0 9.8375 #> 1229 1 3 male 31.0000 0 0 7.9250 #> 1230 0 3 male 27.0000 0 0 8.6625 #> 1231 0 3 female 2.0000 0 1 10.4625 #> 1232 0 3 female 29.0000 1 1 10.4625 #> 1233 1 3 male 16.0000 0 0 8.0500 #> 1234 1 3 male 44.0000 0 0 7.9250 #> 1235 0 3 male 25.0000 0 0 7.0500 #> 1236 0 3 male 74.0000 0 0 7.7750 #> 1237 1 3 male 14.0000 0 0 9.2250 #> 1238 0 3 male 24.0000 0 0 7.7958 #> 1239 1 3 male 25.0000 0 0 7.7958 #> 1240 0 3 male 34.0000 0 0 8.0500 #> 1241 1 3 male 0.4167 0 1 8.5167 #> 1242 0 3 male NA 1 0 6.4375 #> 1243 0 3 male NA 0 0 6.4375 #> 1244 0 3 male NA 0 0 7.2250 #> 1245 1 3 female 16.0000 1 1 8.5167 #> 1246 0 3 male NA 0 0 8.0500 #> 1247 0 3 male NA 1 0 16.1000 #> 1248 1 3 female NA 1 0 16.1000 #> 1249 0 3 male 32.0000 0 0 7.9250 #> 1250 0 3 male NA 0 0 7.7500 #> 1251 0 3 male NA 0 0 7.8958 #> 1252 0 3 male 30.5000 0 0 8.0500 #> 1253 0 3 male 44.0000 0 0 8.0500 #> 1254 0 3 male NA 0 0 7.2292 #> 1255 1 3 male 25.0000 0 0 0.0000 #> 1256 0 3 male NA 0 0 7.2292 #> 1257 1 3 male 7.0000 1 1 15.2458 #> 1258 1 3 female 9.0000 1 1 15.2458 #> 1259 1 3 female 29.0000 0 2 15.2458 #> 1260 0 3 male 36.0000 0 0 7.8958 #> 1262 1 3 female 63.0000 0 0 9.5875 #> 1263 0 3 male NA 1 1 14.5000 #> 1264 0 3 male 11.5000 1 1 14.5000 #> 1265 0 3 male 40.5000 0 2 14.5000 #> 1266 0 3 female 10.0000 0 2 24.1500 #> 1267 0 3 male 36.0000 1 1 24.1500 #> 1268 0 3 female 30.0000 1 1 24.1500 #> 1269 0 3 male NA 0 0 9.5000 #> 1270 0 3 male 33.0000 0 0 9.5000 #> 1271 0 3 male 28.0000 0 0 9.5000 #> 1272 0 3 male 28.0000 0 0 9.5000 #> 1273 0 3 male 47.0000 0 0 9.0000 #> 1274 0 3 female 18.0000 2 0 18.0000 #> 1275 0 3 male 31.0000 3 0 18.0000 #> 1276 0 3 male 16.0000 2 0 18.0000 #> 1277 0 3 female 31.0000 1 0 18.0000 #> 1278 1 3 male 22.0000 0 0 7.2250 #> 1280 0 3 female 14.0000 0 0 7.8542 #> 1281 0 3 male 22.0000 0 0 7.8958 #> 1282 0 3 male 22.0000 0 0 9.0000 #> 1284 0 3 male NA 0 0 7.5500 #> 1285 0 3 male NA 0 0 8.0500 #> 1286 0 3 male 32.5000 0 0 9.5000 #> 1287 1 3 female 38.0000 0 0 7.2292 #> 1288 0 3 male 51.0000 0 0 7.7500 #> 1289 0 3 male 18.0000 1 0 6.4958 #> 1290 0 3 male 21.0000 1 0 6.4958 #> 1291 1 3 female 47.0000 1 0 7.0000 #> 1292 0 3 male NA 0 0 8.7125 #> 1293 0 3 male NA 0 0 7.5500 #> 1294 0 3 male NA 0 0 8.0500 #> 1295 0 3 male 28.5000 0 0 16.1000 #> 1297 0 3 male 27.0000 0 0 8.6625 #> 1298 0 3 male NA 0 0 7.2500 #> 1299 0 3 male 36.0000 0 0 9.5000 #> 1300 0 3 male 27.0000 1 0 14.4542 #> 1301 1 3 female 15.0000 1 0 14.4542 #> 1302 0 3 male 45.5000 0 0 7.2250 #> 1303 0 3 male NA 0 0 7.2250 #> 1304 0 3 male NA 0 0 14.4583 #> 1306 0 3 female NA 1 0 14.4542 #> 1307 0 3 male 26.5000 0 0 7.2250 #> 1308 0 3 male 27.0000 0 0 7.2250 #> 1309 0 3 male 29.0000 0 0 7.8750 ``` ] -- .scroll30[ ```r fare_folds$splits[[1]] %>% assessment() #> survived pclass sex age sibsp parch fare #> 2 1 1 male 0.9167 1 2 151.5500 #> 14 1 1 female 26.0000 0 0 78.8500 #> 16 0 1 male NA 0 0 25.9250 #> 19 1 1 female 32.0000 0 0 76.2917 #> 20 0 1 male 36.0000 0 0 75.2417 #> 25 1 1 female 29.0000 0 0 221.7792 #> 34 1 1 female 58.0000 0 0 26.5500 #> 36 1 1 female 45.0000 0 0 262.3750 #> 41 0 1 male NA 0 0 39.6000 #> 46 0 1 male 45.0000 0 0 26.5500 #> 52 0 1 male 33.0000 0 0 5.0000 #> 53 0 1 male 28.0000 0 0 47.1000 #> 63 0 1 male 46.0000 1 0 61.1750 #> 66 1 1 female 33.0000 1 0 53.1000 #> 70 1 1 female NA 0 1 55.0000 #> 71 0 1 male NA 0 0 0.0000 #> 74 1 1 female 22.0000 0 0 151.5500 #> 80 1 1 female 55.0000 2 0 25.7000 #> 110 1 1 male 36.0000 0 0 26.3875 #> 118 1 1 female 30.0000 0 0 56.9292 #> 129 0 1 male 47.0000 0 0 38.5000 #> 132 1 1 female 45.0000 0 1 59.4000 #> 137 1 1 male 53.0000 0 0 28.5000 #> 142 1 1 female 45.0000 0 1 63.3583 #> 146 1 1 male 48.0000 1 0 76.7292 #> 148 0 1 male NA 0 0 42.4000 #> 152 1 1 male 27.0000 0 0 76.7292 #> 171 1 1 male 49.0000 0 0 0.0000 #> 180 0 1 male NA 0 0 26.5500 #> 198 0 1 male 19.0000 1 0 53.1000 #> 206 0 1 male 65.0000 0 0 26.5500 #> 212 0 1 male 47.0000 0 0 42.4000 #> 231 1 1 female 30.0000 0 0 93.5000 #> 232 1 1 male 52.0000 0 0 30.5000 #> 246 1 1 female 33.0000 0 0 86.5000 #> 262 1 1 male 35.0000 0 0 26.2875 #> 270 0 1 male NA 0 0 26.0000 #> 286 0 1 male 67.0000 1 0 221.7792 #> 292 1 1 female 39.0000 1 1 79.6500 #> 296 1 1 male 17.0000 0 2 110.8833 #> 317 0 1 male 51.0000 0 1 61.3792 #> 321 1 1 male NA 0 0 35.5000 #> 325 1 2 female 28.0000 1 0 24.0000 #> 336 0 2 male 51.0000 0 0 12.5250 #> 355 0 2 male 25.0000 1 0 26.0000 #> 377 1 2 male 24.0000 0 0 10.5000 #> 380 1 2 female 31.0000 1 1 26.2500 #> 426 0 2 male 52.0000 0 0 13.0000 #> 437 1 2 female 45.0000 1 1 26.2500 #> 491 0 2 female 57.0000 0 0 10.5000 #> 498 0 2 male 40.0000 0 0 16.0000 #> 499 0 2 male 32.0000 0 0 13.5000 #> 505 0 2 male 39.0000 0 0 13.0000 #> 507 0 2 male 70.0000 0 0 10.5000 #> 517 0 2 male 36.5000 0 2 26.0000 #> 530 1 2 female 50.0000 0 1 26.0000 #> 541 1 2 female 2.0000 1 1 26.0000 #> 544 0 2 male 36.0000 0 0 10.5000 #> 545 0 2 male 34.0000 1 0 21.0000 #> 549 1 2 male 0.8333 1 1 18.7500 #> 553 0 2 male 19.0000 0 0 10.5000 #> 582 0 2 male NA 0 0 0.0000 #> 585 1 2 female 32.5000 0 0 13.0000 #> 586 0 2 male 27.0000 1 0 26.0000 #> 589 1 2 female 4.0000 1 1 23.0000 #> 593 0 2 male 36.0000 1 2 27.7500 #> 618 0 3 male 25.0000 0 0 7.0500 #> 626 1 3 female 17.0000 4 2 7.9250 #> 633 0 3 female 39.0000 1 5 31.2750 #> 641 0 3 male 9.0000 4 2 31.3875 #> 657 1 3 female 33.0000 3 0 15.8500 #> 669 0 3 male 22.0000 0 0 8.0500 #> 675 1 3 male 32.0000 0 0 56.4958 #> 699 0 3 male 18.0000 0 0 8.6625 #> 701 0 3 male 17.0000 0 0 8.6625 #> 702 0 3 male 17.0000 0 0 8.6625 #> 716 0 3 male 29.0000 0 0 8.0500 #> 728 0 3 male 70.5000 0 0 7.7500 #> 731 0 3 male 27.0000 0 0 7.8958 #> 736 1 3 female 36.0000 0 2 15.9000 #> 737 0 3 male 59.0000 0 0 7.2500 #> 738 0 3 male 19.0000 0 0 8.1583 #> 741 0 3 male 17.0000 0 0 8.6625 #> 756 0 3 male 17.0000 2 0 8.0500 #> 764 1 3 female 0.1667 1 2 20.5750 #> 795 1 3 female 5.0000 0 0 12.4750 #> 803 0 3 male NA 0 0 6.9500 #> 806 1 3 male NA 0 0 56.4958 #> 808 0 3 female 9.0000 2 2 34.3750 #> 836 0 3 male NA 0 0 8.0500 #> 848 0 3 male 23.5000 0 0 7.2292 #> 858 1 3 male 27.0000 0 0 6.9750 #> 867 1 3 female 2.0000 0 1 12.2875 #> 881 1 3 male 29.0000 0 0 7.8958 #> 885 0 3 male 48.0000 0 0 7.8542 #> 893 0 3 male 31.0000 0 0 7.7750 #> 905 0 3 male 23.0000 0 0 7.8958 #> 915 0 3 male 33.0000 0 0 7.8542 #> 921 0 3 male NA 0 0 7.7500 #> 925 0 3 male 34.5000 0 0 7.8292 #> 930 0 3 male NA 1 0 7.7500 #> 941 0 3 male NA 0 0 7.8958 #> 947 0 3 male NA 0 0 56.4958 #> 948 1 3 female 22.0000 0 0 7.2500 #> 959 0 3 female NA 0 4 25.4667 #> 967 0 3 female 25.0000 0 0 7.7750 #> 971 1 3 male 20.0000 1 0 7.9250 #> 975 0 3 male 30.0000 1 0 16.1000 #> 992 1 3 male NA 0 0 7.2292 #> 999 0 3 male NA 0 0 7.8958 #> 1000 1 3 female NA 0 0 7.7500 #> 1013 0 3 female NA 0 0 7.7500 #> 1014 0 3 female NA 0 0 8.0500 #> 1019 0 3 male NA 0 0 8.0500 #> 1029 1 3 female NA 1 0 24.1500 #> 1031 0 3 male NA 0 0 8.4583 #> 1045 1 3 female NA 0 0 15.5000 #> 1090 0 3 male 20.0000 0 0 9.2250 #> 1091 0 3 female 23.0000 0 0 8.6625 #> 1115 0 3 male NA 0 0 7.0000 #> 1127 0 3 female 28.0000 0 0 7.8958 #> 1140 0 3 male 38.0000 0 0 7.8958 #> 1141 0 3 male 21.0000 0 0 8.0500 #> 1157 0 3 female 2.0000 1 1 20.2125 #> 1168 0 3 male NA 0 0 7.7292 #> 1183 1 3 female 21.0000 0 0 7.6500 #> 1195 0 3 male NA 0 0 7.8958 #> 1261 1 3 female 18.0000 0 0 9.8417 #> 1279 0 3 male 20.0000 0 0 7.8542 #> 1283 0 3 male NA 0 0 8.0500 #> 1296 0 3 male 21.0000 0 0 7.2500 #> 1305 0 3 female 14.5000 1 0 14.4542 ``` ] --- class: onecol ## k-fold Cross-Validation in R However, we generally don't need to extract the individual folds. {tune} has built-in functions that can use a `vfold_cv` object directly: - `fit_resamples()` estimates model performance across resamples **with no tuning**. - `tune_grid()` estimates model performance across resamples **with tuning**. -- <p style="padding-top:30px;"> Most ML algorithms include .imp[hyperparameters] to be tuned¹ and require `tune_grid()`. But we can use `fit_resamples()` for 'traditional' statistical models like OLS regression. Luckily, these functions are **nearly identical** so learning one will transfer to the other! .footnote[ [1] We will learn our first ML algorithms and tuning tomorrow! ] --- class: onecol ## Resampling Options There are three possible interfaces to `fit_resamples()` and `tune_grid()`: ```r model_spec %>% fit_resamples(formula, resamples, ...) model_spec %>% fit_resamples(recipe, resamples, ...) workflow %>% fit_resamples(resamples, ...) ``` -- There are also a number of optional arguments, including: - `metrics`: performance statistics to compute¹ - `control`: a list created by `control_resamples()` or `control_grid()` with various resampling options .footnote[ [1] Regression default metrics are RMSE and `\(R^2\)`. Classification default metrics are AUROC and overall accuracy. ] --- class: onecol ## Resampling Options The `control` argument to `fit_resamples()` can be configured by `control_resamples()`¹: Argument | Description :------- | :---------- verbose | Whether to print progress (default = FALSE) save_pred | Whether to save out-of-sample predictions per *k* fold (default = FALSE) event_level | For classification only; specify which level is considered the "event" (`"first"` or `"second"`) extract | An optional function to retain model objects .footnote[ [1] The same list can be created for `tune_grid()` by `control_grid()`. ] --- class: onecol ## Leave-One-Out Cross-Validation The most extreme variation of k-fold CV is when `\(k = N-1\)`. This is called .imp[leave-one-out cross-validation]. A model is trained on `\(N-1\)` rows and used to predict a **single held-out observation**. -- <p style="padding-top:30px;"> The {rsample} package has a `loo_cv()` function that performs LOOCV. However, these objects are not well integrated into the broader tidymodels framework. LOOCV is **computationally expensive** and may have poor statistical properties. -- .bg-light-yellow.b--light-red.ba.bw1.br3.pl4[ LOOCV is not generally recommended. It's usually better to stick with k-fold CV. ] --- ## Comprehension Check \#1 .pull-left[ ### Question 1 **Which is the best data splitting method for ML?** a) A single training and a single test set b) *k*-fold cross-validation c) Repeated *k*-fold cross-validation d) Leave-one-out cross-validation ] .pull-right[ ### Question 2 **What is stratified *k*-fold cross-validation?** a) Each *k* fold contains 1/2 of the data b) Some data are withheld from cross-validation c) Each *k* fold has a similar distribution of data d) Multiple iterations of *k* folds are created ] --- class: twocol ## Should we use a separate test set? -- .pull-left[ k-fold CV is often used on a .imp[full data set]. **Pro**: All data used for training and testing. **Con**: No true test set the model hasn't seen. ] --- class: twocol count: false ## Should we use a separate test set? .pull-left[ k-fold CV is often used on a .imp[full data set]. **Pro**: All data used for training and testing. **Con**: No true test set the model hasn't seen. <img src="data:image/png;base64,#../figs/kfold3.png" width="85%" /> ] -- .pull-right[ Others advocate for a .imp[held-out test set]. **Pro**: Final models are tested on new data. **Con**: Decision-making from a single test set. ] --- class: twocol count: false ## Should we use a separate test set? .pull-left[ k-fold CV is often used on a .imp[full data set]. **Pro**: All data used for training and testing. **Con**: No true test set the model hasn't seen. <img src="data:image/png;base64,#../figs/kfold3.png" width="85%" /> ] .pull-right[ Others advocate for a .imp[held-out test set]. **Pro**: Final models are tested on new data. **Con**: Decision-making from a single test set. <img src="data:image/png;base64,#../figs/kfold4.png" width="110%" /> ] -- .bg-light-green.b--dark-green.ba.bw1.br3.pl4[ **Advice**: A good default is to use 10-fold CV, repeated 3 times, on the entire dataset. ] --- class: onecol ## Should we use a separate test set? <br> ####...all that being said, there *is* an advanced option that allows us to do both! --- class: inverse, center, middle # Nested Cross-Validation --- class: onecol ## Nested Cross-Validation Nested cross-validation adds an additional layer of resampling. This separates the model **tuning**¹ from the model evaluation process. It also frees us from having to rely on a **single** test set to evaluate our model. .footnote[ [1] We'll discuss model tuning in detail tomorrow! ] -- <p style="padding-top:30px;">There are **two layers** of resampling in nested CV. The .imp[outer loop] splits the full data set into not-testing and testing sets. The .imp[inner loop] splits the training data set into model training and validation sets. --- class: onecol ## Nested Cross-Validation For every split of the outer loop, a **full inner resampling split** is conducted. Let's say we use 10-fold CV on the **outer loop** and 5-fold CV on the **inner loop**. This would be a total of .imp[500 models] being fit! -- <p style="padding-top:30px;">In this case, **hyperparameter tuning** is performed within each inner loop. A model is then **fit to each outer split** with the best parameter from that resample. Results are averaged across all outer splits for an **unbiased estimate of the model**. --- ## Nested Cross-Validation .center[ <img src="data:image/png;base64,#../figs/nestedcv_confused.png" width="60%" /> ] --- ## Nested Cross-Validation <img src="data:image/png;base64,#../figs/nested_cv.gif" width="90%" /> --- class: onecol ## Nested CV in R: It's Complicated There is a `nested_cv()` function in {rsample} splits data for nested cross-validation¹. Unfortunately, nested cross-validation is not yet **fully supported** in tidymodels. `fit_resamples()` and `tune_grid()` do not work for nested cross-validation¹. If you want to use nested cross-validation, you will need to write your own functions. .footnote[ [1] Though they work great for regular k-fold CV! ] -- <p style="padding-top:30px;"> Example code can be found on the [tidymodels website](https://www.tidymodels.org/learn/work/nested-resampling/). This is a bit complicated, so **we will stick to using repeated k-fold CV** for this course. But if you are ready to for a challenge, we highly encourage looking into nested CV! --- class: inverse, center, middle # Full Walkthrough of k-fold CV --- class: onecol ## Applied Example: Feature Engineering Let's use `titanic_folds` to fit a resampled model in R predicting `fare`. First, we'll make a preprocessing {recipe} (without prepping or baking). ```r fare_recipe <- titanic %>% recipe(fare ~ .) %>% step_rm(survived) %>% step_naomit(fare) %>% step_mutate(pclass = factor(pclass, levels = c(1, 2, 3)), sex = factor(sex, levels = c("female", "male"))) %>% step_normalize(all_numeric_predictors()) %>% step_dummy(pclass, sex) %>% step_impute_linear(age) %>% step_nzv(all_predictors()) %>% step_corr(all_predictors()) %>% step_lincomb(all_predictors()) ``` --- class: onecol ## Applied Example: Specify Model and Workflow Second, we'll specify a linear regression model. ```r reg_freq <- linear_reg() %>% set_engine("lm") %>% set_mode("regression") ``` -- Third, we'll create a {workflow}. ```r fare_workflow <- workflow() %>% add_recipe(fare_recipe) %>% add_model(reg_freq) ``` --- class: onecol ## Applied Example: Fit the Model Finally, we can fit a resampled model with our {workflow} and `fare_folds`. ```r # configure sampling to save predictions from each k-fold keep_pred <- control_resamples(save_pred = TRUE) fare_results <- fare_workflow %>% fit_resamples(resamples = fare_folds, control = keep_pred) ``` --- class: onecol ## Applied Example: Evaluate a Resampled Model We can evaluate this model with `collect_metrics()`: ```r collect_metrics(fare_results) #> # A tibble: 2 × 6 #> .metric .estimator mean n std_err .config #> <chr> <chr> <dbl> <int> <dbl> <chr> #> 1 rmse standard 38.0 27 2.47 Preprocessor1_Model1 #> 2 rsq standard 0.476 27 0.0179 Preprocessor1_Model1 ``` --- class: onecol ## Applied Example: Evaluate a Resampled Model We can also plot **predicted** against **observed** values to get a better understanding of model performance: .pull-left[ ```r fare_predictions <- collect_predictions(fare_results) fare_predictions %>% ggplot(aes(x = fare, y = .pred)) + geom_point(alpha = .15) + geom_abline(color = 'darkred') + coord_obs_pred() + labs(x = "Observed Fare", y = "Predicted Fare") ``` ] .pull-right[ <img src="data:image/png;base64,#../figs/fare_pred_2c.png" width="70%" /> ] --- class: inverse, center, middle # Time for a Break!